Skip to content

feat: executable conformance — black-box harness + CI validation#16

Merged
trqlmao merged 4 commits into
mainfrom
feat/conformance-suite
Jun 8, 2026
Merged

feat: executable conformance — black-box harness + CI validation#16
trqlmao merged 4 commits into
mainfrom
feat/conformance-suite

Conversation

@trqlmao

@trqlmao trqlmao commented Jun 8, 2026

Copy link
Copy Markdown
Owner

Draft for review. Stacked on #15 (base = docs/impl-ergonomics) — merge #15 first, then this retargets to main automatically. Diff shown here is only the conformance work.

Turns "conformant" from self-attested into executable, and wires the machine-readable artifacts so they can't silently drift.

What's in it

Change File(s) Why
Black-box conformance harness harness/ Point it at any running server → drives the full wire contract, asserts the normative MUSTs (not just a transcript). The thing real standards have (OIDC/ACME).
Conformance CI .github/workflows/conformance.yml Runs the vector runner (no CI until now), Spectral-lints openapi.yaml, checks schema/example/index/openapi consistency, and runs the harness against the Go + TypeScript reference servers.
Schema-consistency tests examples/conformance/test/schema.test.ts (+ ajv, yaml) Every examples/*.json validates against its schema $def; vectors/index.json lists exactly the files present; every openapi.yaml $ref resolves.
Spectral ruleset .spectral.yaml OpenAPI lint config.

Harness coverage (24 checks)

  • §3 auth — nonce shape; valid token; single-use nonce rejection; the classic "signed the base64 string, not the raw nonce" rejection; 401 without a token.
  • §6 vault — createRepo sole-member→403 / duplicate→409; pull unchanged vs envelope; push accept + version bump; stale push = 200 conflict, never 4xx; non-member→403; addMember/fetchMemberKey; removed-member→403.
  • §4/§10 — envelope round-trips to our plaintext; a 2nd member unwraps + decrypts; zero-knowledge (plaintext never appears in the server's JSON or stored ciphertext); removeMember bumps epoch; departed member's old key can't decrypt the rotated envelope; a remaining member can.

The only AVP-specific code is the client crypto, reused from the vector-tested examples/typescript/client — the harness stays a thin wire driver.

Verified locally

  • Harness: 24/24 PASS against the TypeScript reference server (bun harness/conformance.ts).
  • examples/conformance bun run test: 24 tests pass (vectors + new schema/example/index/openapi checks); typecheck clean.
  • harness tsc --noEmit: clean (cross-dir import resolves).
  • Spectral: 0 errors (4 non-failing "missing description" warnings).
  • no-leak scan clean on every changed file.

Go-server CI path couldn't be run locally (no Go toolchain here) — it reuses the proven server-start/poll pattern from examples-interop.yml with SHA-pinned actions; CI is the check.

Still open (from the menu, not in this PR)

Reference IdP + persistent server; federation vectors (invite/locator base64url, avp:// URI); THREATMODEL.md; vector generator script; negative/"MUST reject" vector bank; gRPC example; browser/WASM demo; llms-full.txt; spec quardrails (size limits, idempotency, Retry-After).

trqlmao added 4 commits June 8, 2026 10:40
Close the gap between the turnkey crypto path and the wire/transport
layer, where most "get it right first try" friction lives for both
humans and LLMs implementing AVP.

- openapi.yaml: OpenAPI 3.1 description of the HTTP/JSON profile — every
  path, method, status code, and error response — referencing the JSON
  Schema shapes so the route surface is machine-readable and
  stub-generatable. The gRPC profile had proto/avp.proto; HTTP had only a
  prose route table.
- schema + SPEC §6: define the error body { error, code?, detail? } and a
  status→code table, and state that every non-2xx is terminal while a
  stale-version push is a 200 with conflict:true. The five reference
  servers already emit { error, detail? }; `code` is additive, so they
  stay conformant.
- SPEC §11: reword conformance to map each requirement to the vector that
  actually exists (Ed25519 anchors the challenge signature; the
  payload-AEAD epoch-tamper case anchors rotation correctness) instead of
  implying standalone challenge/token and rotation vector files that do
  not exist. Dedicated vectors for those paths are noted as welcome
  additions.
- vectors/index.json: machine-readable index of the vectors (file, kind,
  spec section, RFC anchors) so a harness can enumerate them without
  hardcoding filenames.
- Taskfile.yml: `task test` runs the conformance suite and every
  language's example tests in one command, mirroring the CI jobs.

Cross-linked from README, IMPLEMENTING.md, llms.txt, examples/README.md,
and vectors/README.md; recorded under CHANGELOG [Unreleased].
Make "conformant" executable rather than self-attested, and stop the
machine-readable artifacts from silently drifting apart.

harness/ — a black-box conformance suite. Point it at any running AVP
server (`bun harness/conformance.ts --server URL`) and it drives the full
HTTP/JSON wire contract, asserting the normative MUSTs instead of just
printing a transcript:
- §3 auth: nonce shape, valid-token, single-use-nonce rejection, the
  classic "signed the base64 string not the raw nonce" rejection, 401
  without a token.
- §6 vault: createRepo sole-member/duplicate rules, pull unchanged vs
  envelope, push accept + version bump, a stale push as a 200 conflict
  (never a 4xx), non-member 403, addMember/fetchMemberKey, removed-member
  403.
- §4/§10 crypto + invariants: envelope round-trips to our plaintext; a
  second member unwraps and decrypts; zero-knowledge (the plaintext never
  appears in the server's JSON or stored ciphertext); removeMember bumps
  the epoch and a departed member's old key cannot decrypt the rotated
  envelope; a remaining member can.
The only AVP-specific code is the client crypto, reused from the
vector-tested examples/typescript/client, so the harness stays a thin
wire driver. Verified locally: 24/24 against the TypeScript reference
server.

.github/workflows/conformance.yml — runs the vector runner (which had no
CI until now), lints openapi.yaml with Spectral, checks schema/example/
index/openapi consistency, and runs the harness against the Go and
TypeScript reference servers (SHA-pinned actions, the interop workflow's
server-start/poll pattern).

examples/conformance — a new test asserts every examples/*.json body
validates against its schema $def, that vectors/index.json lists exactly
the vector files present, and that every openapi.yaml $ref resolves to a
schema $def (adds ajv + yaml dev-deps).

Cross-linked from README, IMPLEMENTING.md (§6 checklist), SPEC.md (§11),
llms.txt, and Taskfile.yml (`task harness`, `task openapi-lint`);
recorded under CHANGELOG [Unreleased].
CLAUDE.md rule: no em dashes. Replace the em dashes introduced with the
openapi/error/index/harness work (and one pre-existing in SPEC §6) with
commas, colons, semicolons, or parentheses. No semantic change.
Add openapi.yaml, the HTTP error body ($defs/Error), vectors/index.json,
the harness/ black-box suite, the Taskfile runner, and the conformance CI
workflow to the AI-agent mental model; list Go among the examples; extend
the "one protocol, keep in sync" note to cover openapi.yaml.
@trqlmao trqlmao changed the base branch from docs/impl-ergonomics to main June 8, 2026 18:59
@trqlmao trqlmao marked this pull request as ready for review June 8, 2026 18:59
@trqlmao trqlmao merged commit 35e1ec4 into main Jun 8, 2026
32 checks passed
@trqlmao trqlmao deleted the feat/conformance-suite branch June 8, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant